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ABSTRACT 



This research explored the degree to which group 
sizes can differ bef,.ore the robustness of analysis of variance 
(ANOVA) and analysis of covariance (ANCOVA) are jeopardized, Monte 
Carlo methodology was used, allowing for the experimental 
investigation of potential threats to robustness under conditions 
common to researchers in education. The effects of unequal group 
sizes w*re explored under the following data set conditions: (1) 
heterogeneity of group variances; (2) skew; (3) kurtosis; and (4) in 
ANCOVA, heterogeneity of regression slopes. Two independent sets of 
simulations were conducted, one using a total group of 90 and the 
other a total of 60. Experimentation was limited to simulations using 
three groups, with the total number divided in a systematic fashion. 
Results of these studies produced results consistent with previous 
research. In the analyses having homogeneity of group variances, the 
only simulations that emerged as statistically significant from the 
theoretical F test were those that had a large degree of difference 
in group numbers and unequal regression slopes. No significant 
differences emerged in simulations with near equal numbers. Tables in 
this paper document the simulation results, but also offer the 
research practitioner some idea of the true risk of Type I error in 
such situations. (Contains 7 tables and 24 references.) (SLD) 
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INTRODUCTION 



The analysis of variance (ANOVA) statistical procedure has been found to be the most 
widely used technique in several major educational journals, and along with its logical extension - 
the analysis of covariance (ANCOVA) - is widely used in psychology and the social sciences as 
well (Elmore and Woehlke, 1988; Goodwin and Goodwin, 1985; Halpin and Halpin, 1988). Like 
other statistical procedures derived from the general linear model, ANOVA and ANCOVA are 
designed to produce results that are both statistically robust and powerful when specific 
mathematical conditions (i.e., statistical assumptions) are present. However, practitioners seldom 
find the ideal research situation where all of these conditions are met. 

Previous research has, for example, found that the robustness of both the ANOVA and 
ANCOVA F is questionable in face of unequal group sizes and/or heterogeneity of group 
variances (e.g., Box, 1954; Glass et al., 1972; Johnson and Rakow, 1993; Scheffe', 1959) - 
problems that researchers in these disciplines are often unable to avoid. Thus, the critical 
question that arises is how much difference exists between the conditions these statistical 
procedures were designed to handle and the actual conditions that occur in a particular 
researcher's situation. If the difference is within a "tolerable range," then use of ANOVA or 
ANCOVA should provide information that is statistically robust. However, when the data collected 
exceed that "tolerable range," alternatives to these parametric procedures must be considered. 

Theoretical mathematics can be used to define the general nature of the problems that 
arise when ANOVA or ANCOVA are used inappropriately, however it is unable to provide us with 
the precise limits of this "tolerable range." Therefore, this research uses Monte Carlo 
methodology to provide supplemental information which cannot be obtained using theoretical 
mathematics. Using Monte Carlo methodology, it is possible to numerically define the degree of 
tolerance (i.e., robustness) that ANOVA and ANCOVA have in practice. 

Thft <frv»infthis Research 
This research is designed to examine the effects of unequal group sizes on the 
robustness of the ANOVA and ANCOVA statistical procedures. Specific questions this research 
is designed to address include the following: (1) How different can the group sizes be without 
jeoparciizing the results? (2) Does the "tolerate range" in group sizes change when violations of 
data set assumptions are also present? (3) How do unequal n's affect the true level of 
significance (that is, the true risk of making type I error) in a given research situation? 

REVIEW OF THE LITERATURE 

The t-test for two independent variables tests for mean differences between two groups. 
The oneway, fixed-effects ANOVA Is a logical extension of this t-test, testing for mean differences 
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between two or more groups. In oneway ANOVA, F tests the ratio of explained variance in the 
dependent variable relative to the variance left unexplained. The one concomitant (i.e., one 
covariate) ANCOVA procedure is a logical extension of oneway ANOVA, applicable when a third, 
continuous variable is known to have a significant effect on the dependent while having little or no 
effect on the independent variable. When ANCOVA is appropriate, the researcher's goal is to 
probe the effects of the independent variable on the dependent variable arter removing the 
influence of the concomitant. To do this, ANCOVA first removes all variation in the dependent 
variable that is a function of the concomitant. Using these "adjusted scores," ANCOVA then 
effectively reanalyzes the data for mean differences between the groups. 

According to Cochran (1957) and Winer (1962), the relationships found in the simple 
oneway ANOVA and even the simpler West for two independent samples carries over to 
ANCOVA. Therefore, this literature review will contain discussion of relevant theoretical and 
empirical research involving ail of these procedures. 

statistical Procedures and the Assumptions Inherent Within Them 
Statistical procedures are developed by mathematicians to be used under a specific set of 
mathematical conditions (i.e., assumptions about the data set being used). These conditions are 
designed to balance creditability (the ability to process data in a usable form) with manageability 
(the technique's ability to simplify many mathematical derivations and operations). Valid results are 
obtainable when a researcher's data set adheres to these assumptions. 

However, seldom do data sets adhere perfectly to these assumptions. According to 
Glass, Peckham and Sanders (1972), the question that a researcher must ask in reference to the 
data collected is not tvftetterthe assumptions are satisfied, but instead, are the violations that do 
occur extreme enough to compromise the results? Box and Anderson (1955) note that to fulfill 
the needs of the researcher, statistical criteria should (a) be sensitive to changes in the specific 
factors being tested (i.e., they should be powerful) and (b) they should bcj insensitive to changes 
in extraneous factors of a magnitude likely in practice (i.e., they should be robust). 

Literature Concerning the Qnewav Analysis of Variance 

Homogeneity of Variance 

This assumption was first identified in the classical 1908 paper "The Probable Error of the 
Mean" by The Student (Gossett); however the publishing of empirical results in this area would 
wait until Hsu (1938, as cited by Scheffe', 1959). Many published studies suggest the F test with 
equal sample n's is robust when faced with the single violation of unequal group variances as long 
as the ratio of largest to smallest group variances does not exceed three (e.g., Glass et al., 1972). 
However, empirical research by Johnson and Rakcw (1993) found balanced ANOVA results to be 
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robust even when the ratio of largest to smallest group variance was five. F is not robust, 
however, when both group sizes and variances are unequal. When n's are unequal and two 
groups used. Inflated type I error rates occur when the larger group Is paired with the smaller 
variance (Box, 1954; Glass et al. 1972; Johnson and Rakow, 1993; Shields, 1978). 
Furthermore, Tomarken and Sertin (1986) have argued that ANOVA may not be the best choice in 
the presence of heterogeneity of variances, especially when many groups are to be compared. 
They suggest that th« effects of variance heterogeneity increases with the number of groups. 

Nfir rpanty °f Ihft Dlstti&UflBD of Errors 

Research dating back seven decades has investigated violations of this assumption. 
Perhaps among the most notable is the work of Games and Lucas (1966). who suggest that 
skewed distributions are a greater threat to robustness than leptokurtlc or platykurtic distributions. 
Furthermore, they suggest that F test results may improve some for highly leptokurtlc error 
distributions. Johnson and Rakow (1993). however, found that distributional shape did not have 
a significant effect on the robustness of ANOVA with equal n's. 

F xtftnsinn o f am nvA Asstimntions to ANCOVA 

The simplest form of ANCOVA (which consists of one independent, one concomitant and 
one dependent variable) is an extension of oneway ANOVA. According to Cochran (1957) and 
Winer (1962), the assumptions previously discussed in regards to ANOVA apply to ANCOVA as 
well, provided the concomitant variable is normal. 

Thn ft f w an Assum ption* nf the Analysis of Qoygriance 
Elashoff (1969) and McLean (1979, 1989) reported the following seven assumptions 
associated with ANCOVA: (a) the cases are assigned at random to treatment conditions; (b) the 
covariate is measured error-free (that is, perfect reliability of the covariate measurement); (c) the 
covariate Is independent of the treatment effect; (d) the covariate has a high correlation with the 
dependent variable; (e) the regression of the dependent variable on the covariate Is equal for 
each group; (f) for each level of the covariate, the dependent variable is normally distributed; and 
(g) the variance of the depend^ ,rt variable at each value of the covariate is constant across 
groups. These assurr^tons can be dassifl^ ( a > 
assumptions concerned with research design and sampling (methodological assumptions) and (b) 
assumptions concerned with the the data set and its parent population (data set violations). 

Mnthmlrilnglfi?' Assumptions 

Two assumptions deal with research design and sampling: (a) the cases are assigned to 
randomtreatments and (b) the covariate has perfect reliability. Concerning randomization. Evans 
and Anastaslo (1968) distinguished three separate sceneries: (a) Individuals are assigned to 
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groups at random after which treatments are randomly assigned to the groups; (b) intact groups 
are used, but treatments are randomly assigned; and (c) intact groups are used where treatments 
occur naturally rather than being randomly assigned by the researcher. They maintain that 
ANCOVA is appropriate for the first situation, can be used with caution in the second, but should 
be abandoned altogether (perhaps in favor of the less restraining factorial block ANOVA ) in the 
third. Two reasons are provided for their recommendations: first, ft is never clear whether the 
covariance adjustment has removed all of the Was when proper randomization has not taken 
place, and second, when there are real differences among groups, covariance adjustments may 
involve computational extrapolation. 

A number of researchers (e.g., Loftin and Madison, 1991 ; McLean, 1974; Raajimakers 
and Pieters, 1987; and Thompson, 1992) have addressed the issue of an unreliable covariate. 
Raajimakers and Pieters noted that there are two ways that a researcher can conceptualize 
covariate reliability. If one assumes the dependent is linearty related to the observed value of the 
covariate, then ANCOVA retains statistical validity. If, on the other hand, ft is assumed that the 
dependent is linearly related to the underlying true score on the covariate (rather than the sample 
that was actually observed), then the F ratio will produce biased results. McLean's research, 
however, suggest that the issue of perfect reliability is less of a threat to the reliability of the F ratio 
if there is an independence of the covariate measure across treatment groups. 

Thp nnvariptft's Relationship with the In dependent and Dependent Variables 

The covariate should be highly correlated with the dependent variable, yet should not be 
correlated with the independent. Feldt (1958) recommended use of a covariate only when the 
zero-order correlation between the covariate and dependent variable is r ;> 0.6. McLean (1979, 
1989) saw the relationship between the covariate and independent variables to U> the most 
fundamental of assumptions, suggesting that ANCOVA not be performed until after the data has 
been tested to see if it meets this assumption. If it is not met, the results are not invalidated as 
such, however it reduces ANCOVA's efficiency to slightly below that of ANOVA on the same data. 

Hnmng ft nRlt Y nf Grou p Regression Slopes 

Thib assumption requires that the slope of the regression line between the concomitant 
and dependent variables be the same for all levels of the grouping variable (see McLean, 1979, 
1989; Thompson, 1992). The problem, if this assumption is violated, is analogous to interpreting 
main effects in the presence of significant interactions in an n-way factorial ANOVA. If 
heterogeneous slopes are suspect, the rano> nized block ANOVA is preferable to ANCOVA. 

McClaren (1972) , Shields, (1978), and Johnson and Rakow (1993) empirically 
investigated the effects of violation of this assumption using unequal group sizes. These studies 
suggest that when the smallest regression coefficient and the largest variance are combined with 
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the smallest sample size, the empirical significance levels are biased in a non-conservative 
direction, and likewise, when the pairings are reversed, the 'est becomes conservative. 

RESEARCH METHODOLOGY 

This researjh explored the degree to which group sizes can differ before the robustness 
of analysis of variance (ANOVA) and analysis of covariance (ANCOVA) are jeopardized. Monte 
Carlo methodology was used, allowing for experimental investigation of potential threats to 
robustness under conditions common to researchers in education. Specifically, this research 
explored the effects of unequal group sizes with the following data set conditions: heterogeneity 
of group variances, skew, kurtosis, and (in ANCOVA) heterogeneity of regression slopes. 

J hfl SamQUtiog Environment an ri Programs Written tr> Conduct the Simulations 
The statistical simulations were conducted on a VAX 6430 mainframe computer. A 
custom designed simulation package written in FORTRAN 77 was used, employing mathematical 
subroutines from the International Mathematical and Statistical Libraries (IMSL). 

Two independent sets of simulations were conducted: one using a total group n of 90 
and the other one using a total group n of 60. Experimentation was limited to simulations ui'ing 
three groups, and the total n was divided among the three groups in a systematic fashion 
designed to represent group size combinations similar to those faced by educational researchers. 

Tfifl Simulation PrnoflfiR Part I: W ithin a Single Replication of an Experiment 

The data were ck ited using the International Mathematical and Statistical Libraries (IMSL) 
subroutine RNVMN, which is designed to create multivariate nomai distributions with means of 
zero, variances of one, and correlations between vectors that can be specified by the user. Three 
vectors were created by IMSL: one dependent and two concomitant. Data for each treatment 
level were created separately and later merged, making it possible to obtain the unequal group 
slopes desired for the second concomitant vector. For the first concomitant, correlation between 
all groups and the IMSL created dependent vector was set at r - .707, thus simulating 
homogeneity of regression slopes. For the second concomitant, heterogeneity of slopes was 
simulated by creating concomitant vectors for group 1 that had a correlation of r - .6 with the group 
1 dependent vector, a con elation of r - .707 between the second group concomitant and 
dependent vectors and r - .8 between the third group's concomitant and dependent vectors. 

IMSL created dependent vectors which were normal around the first and second 
moments (i.e., "beli-shaped"). Since the integrity of the data generating process depended on 
these vectors being mesokurtic and with normal skew, a testing procedure was built into the 
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computer program to screen out vectors with skew and kurtosis that fall outside the 95% 
confidence intervals surrounding zero skew and kurtosis. Once an acceptable dependent 
vector was created, six additional copies were made and then perturbed so as to create vectors 
with the following shapes: platykurtic (skew « 0, kurtosis - -.5), leptokurtic (skew - 0, kurtosis « 
2), moderate skew (skew - .5, kurtosis - 0), moderate skew and platykurtic (skew - .5, kurtosis - 
-.5), moderate skew and leptokurtc (skew « .5, kurtosis - 2), and extremely skewed and 
leptokurtic (skew - 1 , kurtosis - 2). After perturbation, three duplicate copies of each of these 
seven vectors were created, and linearty transformed to create vectors with one of four degrees of 
variation: homogeneity of group variance (1 :1 :1), slight heterogeneity of variance (1 :1/2:2), 
moderate heterogeneity of variance (1 :2:3), and extreme heterogeneity of variance (1 3:5). 

In the end 28 different dependent vectors, two concomitants and a grouping vector were 
created. For the ANOVA simulations, the grouping vector was combined with each dependent 
vector, resulting in 28 F ratios - one for each combination of skew, kurtosis, and variance. For 
ANCOVA, the first concomitant (with equal slopes) was combined with each of the 28 dependent 
vectors, then the second concomitant (with unequal slopes) was combined with each of the 28 
dependent vectors. This produced an additional 56 ANCOVA F ratios. 

In addiJion to using IMSL subroutines to generate the data, IMSL was also used to 
calculate the F ratios. Specifically, IMSL subroutine AONEW was used to obtain the ANOVA F 
values, while AONEC was used to calculate the ANCOVA F values. 

The Simulation Process. Pa rt II: Tha Global Design 
As has been mentioned previously, two sets of analyses were run: one set containing 
simulations of three groups with a total n of 90 and the other set using simulations of three groups 
with a total n of 60. These sample sizes were chosen because they are similar to what educational 
researchers often encounter in practice. The use of two Independent sets of simulations was 
done to allow for the cross-validation of data patterns that might be identified by the research. 

For each of the two sets of analyses, the number assigned to each of the three groups 
was allowed to vary - thereby producing degrees of difference in the unequal n designs. When 
the total n was 90, the following combinations were tested: 30 in all three groups; groups of 28, 
30 and 32; groups of 26, 30 and 34; groups of 24, 30 and 36; groups of 22, 30 and 38; and 
groups of 20, 30 and 40. These group size combinations resulted in the following percentages of 
difference between the largest and smallest group sizes: 0%, 12.5%, 24%, 33%, 42%, and 50%. 
When the total n was 60, the following combinations were tested: 20 in all three groups; groups 
of 18, 20 and 22; groups of 16, 20 and 24; groups of 14, 20 and 26; and groups of 12, 20 and 28. 
These gruup sizes resulted in the following differences: 0%. 18%, 33%, 46%, and 57%. 
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Each of these group size combinations were combined with one or more of the following 
data set violations: skew, kurtosis, heterogeneity of group variances and (in ANCOVA) 
homogeneity of regression slopes. This allowed for the identification of factors that threaten the 
robustness of ANOVA and ANCOVA, and allowed the determination when each of these factors 
should become a source of concern for the research practitic ner. 

Glass et al. (1972) recommended that sampling distributions created In Monte Carlo 
studies have a minimum of 2000 F ratios each. In those simulations involving homogeneity of 
group variances (1 :1 :1), the number of data points in the san pling distribution were 4000 - twice 
the recommendation of Glass et al. Where unequal n's were combined with heterogeneity of 
variances, sampling distributions of 2000 F ratios were created; each representing one of two 
experimental conditions which previous literature (e. g., Box, 1i>54; Glass, 1972) suggest will 
cause differing effects on type I error rates. For ANOVA, condition A paired the smallest n with the 
smallest variance, while the largest n was paired with the largest valance. Condition B paired the 
smallest n with the largest variance, while the largest n was paired witi: the smallest variance. For 
ANCOVA, condition A occured when the smallest n was joined with the smallest regression 
coefficient and the smallest variance, while the largest n was joined with the largest regression 
coefficient and the largest variance. Similarly, condition B occured when the smallest n was joined 
with the smallest regression coefficient and the largest variance, while the largest n was joined 
with the largest regression coefficient and the smallest variance. 

After all simulations were run, 462 empirical sampling distributions of 4000 F ratios each 
were created, representing all single and compound violations for ANOVA and ANCOVA when 
homogeneity of variance was present. Another 2772 sampling distributions of 2000 F ratios each 
Included all ANOVA and ANCOVA simulations with heterogeneity of variances present. 

Of these 3234 F sampling distributions, eleven ANOVA and eleven ANCOVA F sampling 
distributions (one for each sample size combination) were created using data having no violation 
under study. These twenty-two sampling distributions served as a baseline for comparisons as 
well as also providing a check to make sure the simulation was operating property. 

Statistical Analysis of the Sampling Distributions 
Data were analyzed using three different methods: (1 ) significance testing of sampling 
distribution tails (the area of the curve where hypothesis testing decisions are made), (2), through 
the comparison of nominal alpha levels with the actual (observed) significance levels found in the 
simulation process, and (3) graphically. Significance testing was performed using the 
Kolmogorov-Smlmov one sample test at the a < .05 and (where applicable) a < .01 levels of 
significance. This procedure allows comparison of an empirical data distribution with a standard 
distribution. Specific to this research, the tall regions of each of the 3234 F sampling distributions 
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were compared against the tall region of the appropriate theoretical F sampling distribution. 
Significant differences suggest a lack of robustness for the single or compound violation tested. 

In addition to statistical testing, comparisons of the actual significance levels found in the 
simulations were compared against the nominal statistical level (that is, the statistical level a 
researcher believes he or she is using). The three nominal statistical levels examined in this 
research are a < .1 0, a < .05 and a < .01 . It is customary for a researcher to choose one of 
these significance levels in his or her own hypotheses testing - accepting at faith the fact that it 
represents the true probability of making type I error. The actual significance level, on the other 
hand, represents a more accurate estimate of the probability of making type I error when faced 
with a specific set of mathematical conditions. 

RESULTS 

Summarized below are the effects of violations of data set assumptions for the ANOVA 
and ANCOVA simulations included in this study. Since the integrity of the results is dependent 
on the quality of the data generated, the first section will discuss the population data. The second 
section will discuss simulations with homogeneity of variances. The next two sections will 
summarize the results of ANOVA and ANCOVA simulations with heterogeneity of variances. 
Finally, a summary of the results of the graphic analysis is included. 

The results discussed beiow air: analyzed using three different methodologies. The first 
method uses the Kolmogorov-Smirnov test on each of the 3234 sampling distributions. The KS 
test compares the tail region of each empirical sampling distribution against the appropriate 
theoretical (I.e., nominal) F distribution to determine If the empirical distribution tail is significantly 
different from theoretical F. The second method compares the observed significance levels (i.e., 
the observed type I error rate) for each of the 3234 mathematical conditions against the nominal 
alpha level (i.e., the expected type I error rate). To conserve space, separate KS tables are not 
included here. Instead significance testing results are color coded into the actual significance 
level tables. Finally, the third method involves examination of the data graphs. 

Analysis nf tho Po pulation Data 
Data created in each replication were retained in order to verify the integrity of the results. 
Vectors for each treatment level were created Individually. Later they were merged with vectors 
for the other treatment levels so thst ANOVA and ANCOVA could be performed. Population 
vectors were checked twice: once for each treatment level separately, and again after treatment 
levels were merged. All population vectors, including both those created originally by IMSL and 
those that were the result of algebraic transformation, were at or near the target parameters. 
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Population sizes are worth noting. In their classic 1972 paper, Glass et al. suggested that 
populations with the desired characteristics have a minimum 10,000 data points each. Population 
N's used in this study were considerably larger than that minimum: for the full (merged) vectors 
having homogeneity of variance, the N's were 360,000 when the total experiment n was 90 and 
240,000 when the total n was 90. For full (merged) vectors having heterogeneity of variance, the 
N's were 180,000 when n was 90 and 120,000 when n was 60. 

Effects of Data Set Violations When Homogene ity nf Variance is Present 

ftesntts When the Total n « 90 

In the absence of heterogeneity of group variances, no significant differences emerged 
between the nominal F distribution and the empirical F sampling distributions until the magnitude 
of group size differences became 50% {the 20:30:40 simulation). When group n's became this 
extreme, KS testing revealed that all ANCOVA simulations having unequal group regression 
slopes were significantly different from the theoretical F distribution at the a < .01 level. 

With a nominal significance (i.e., alpha) level of a < .10, a researcher expects 10% chance 
of making type I error. Likewise, at nominal significance levels of a < .05 and a < .01 , the 
research practitioner expects 5% and 1% chance of type I error respectively. When homogeneity 
of variance is present, the actual (i.e., observed) differences for ANOVA simulations as well as all 
equal slope ANCOVA simulations were found to be within ±.01 of the expected levels at <x< .10 
and a < .05, and within ± .003 at a < .01 . This was true even when the difference between the 
largest and smallest group size was as large as 50%. 

When ANCOVA involved unequal regression slopes, on the other hand, larger 
differences from theoretical alpha emerged when group n's were very unequal. For the 22:30:38 
simulation (where the groups differed by 42%), the observed risk of type ! error was found to be 
between 9% and 1 3% (i.e., ±.02 of nominal alpha) for the nominal level of a < .1 0. When the 
group sizes were 20:30:40, an even larger discrepancy emerged at the a < .10 level (±.04 of 
nominal alpha). Observed error rates for a < .05 fell between .05 and .07, while for a < .01 they 
were between .008 and .022 (see table 1). 

Results Whe n the Total n = 60 

In the absence of heterogeneity of group variances, no significant differences emerged 
between the nominal and empirical F distributions until the group size differences became 57% 
(the 12:20:28 simulation). When group n's were this extreme however, KS testing revealed that 
leptokurtic ANCOVA simulations having unequal slopes were significant at the a < .05 level. 

At the nominal level of a < .10, the observed significance levels for ANOVA and those 
ANCOVA's with equal slopes varied ±.01 from nominal alpha , no matter what the degree of 
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Tabu 1: Actu«t Significant Umli {^Hnmngtnrlty of Yirlinct it Prtitnt 



Group Size Comblnetlone 90 : 30 : 30 29 : 30 : 32 

Nprrynal Alpha Levels p<10 p<L05 p<01 p<,1CL 



2$ : 30 : 34 24 : 30 36 22 : 30 : 33 20 : 30 : 40 

P<1Q p<05 p<Q1 p<.1Q p<05 p<.Q1 p<lO p<05 p<.01 p< 10 p<.0S d<01 



.10 


.05 


.009 


.11 


.05 


.012 


.09 


.05 


.008 


.10 


.05 


.014 


.10 


.05 


.009 


.10 


.05 


.013 


.K 


.05 


.009 


.10 


.05 


.012 


.09 


.05 


.008 


.10 


.05 


.014 


.09 


.05 


.008 


.10 


.05 


.013 


.10 


.05 


.012 


.11 


.06 


.012 


.10 


.04 


.008 


.10 


.05 


.013 


.10 


.05 


.009 


.11 


.05 


.012 


.10 


.05 


.009 


.10 


.05 


.012 


.10 


.04 


.008 


.10 


.05 


.015 


.10 


.04 


.010 


.10 


.05 


.013 


.09 


.04 


.008 


.10 


.05 


.012 


.09 


.05 


.008 


.10 


.05 


.014 


.09 


.04 


.009 


.09 


.04 


.013 


.10 


.05 


.008 


.11 


.05 


.012 


.10 


.05 


.010 


.10 


.05 


.013 


.10 


.05 


.010 


.11 


.05 


.012 


.10 


.04 


.009 


.10 


.06 


.011 


.10 


.05 


.010 


.10 


.05 


.014 


.10 


.05 


.011 


.10 


.05 


.012 



.10 


.05 


.009 


.10 


.05 


.013 


.10 


.05 


.010 


.10 


.05 


.01 1 


.09 


.05 


.009 


.10 


.05 


.013 


.10 


.05 


.009 


.10 


.05 


.013 


.10 


.05 


.010 


.10 


.05 


.012 


.09 


.05 


.009 


.10 


.05 


.013 


.10 


.06 


.012 


.10 


.05 


.012 


.*0 


.05 


.010 


.10 


.05 


.011 


.10 


.05 


.009 


.11 


.05 


.012 


.10 


.05 


.009 


.10 


.05 


.013 


. !0 


.05 


.010 


.11 


.05 


.010 


.09 


.05 


.007 


.10 


.05 


.013 


.09 


.05 


.009 


.10 


.05 


.013 


.10 


.05 


.010 


.10 


.05 


.009 


.09 


.04 


.006 


.09 


.04 


.013 


.10 


.06 


.008 


.10 


.05 


.012 


.10 


.05 


.010 


.10 


.05 


.011 


.09 


.04 


.009 


.11 


.05 


.012 


.10 


.06 


.009 


.10 


.05 


.012 


.10 


.05 


.010 


.10 


.05 


.010 


.09 


.04 


.008 


.10 


.05 


.012 


zQUAL 


REGRESSION 


SLOPES 





























A NOV A 

Distribution*! ShMfiM 

Normal 
Platykurtc 
Leptokurbc 
Moderate Skew 
Mod Skew. & Ptaty 
Mod. Skew & Lepto 
Ext Skew & Lepto 



ANCOVA: WITH EQUAL REGRESSION SLOPES 
Oittfibutionti Shift* 



Norma) 
Platykurtjc 
Leptokurtic 
Moderate Skew 
Mod Skew* Ptaty 
Mod. Skew & Lepto 
Ext Skew & Lepto 

ANCOVA : WITH 
Diatributiona! Snap* 

Normal 
Platykurbc 
Leptokurtic 
Moderate Skew 
Mod Skew & Ptaty 
Mod. Skew & Lepto 
Ext Skew & Lepto 



A NOV* 

DiatribuHonal Shapa 

Normal 
Platykurtc 
Leptokurtic 
Moderate Skew 
Mod Skew & Ptaty 
Mod. Skew & Lepto 
Ext Skew & Lepto 



ANCOVA : WITH EQUAL REGRESSION SLOPES 
Diatributlonai Snap* 



Norma) 
Platykurtc 
Leptokurtic 
Moderate Skew 
Mod Skew & Ptaty 
Mod. Skew & Lepto 
Ext Skew & Lepto 



ANCOVA : WITH UNEQUAL REGRESSION SLOPES 
Dimtributlonal Shape 



Norma! 
Piatykurtic 
Leptokurtic 
Moderate Skew 
Mod Skew 4 Pltty 
Mod. Skew & Lepto 
Ext Skew & Lepto 



.09 


.05 


.008 


.10 


.05 


.013 


.10 


.05 


.012 


.12 


.06 


.022 


.12 


.07 


.016 


.13 


.07 


.017 


.09 


.05 


.008 


.10 


.05 


.013 


.10 


.05 


.013 


.11 


.06 


.022 


.13 


.07 


.016 


.14 


.07 


.016 


.10 


.05 


.012 


.11 


.05 


.012 


.11 


.05 


.012 


.12 


.06 


.021 


.13 


.07 


.016 


.14 


.07 


.015 


.10 


.05 


.009 


.10 


.05 


.013 


.10 


.05 


.013 


.12 


.06 


.017 


.13 


.07 


.016 


.14 


.07 


.016 


.10 


.05 


.008 


.10 


.05 


.013 


.10 


.05 


.012 


.11 


.06 


.017 


.13 


.07 


.016 


.14 


.07 


.015 


.10 


.05 


.008 


.11 


.05 


.012 


.10 


.05 


.013 


.11 


.06 


.018 


.13 


.07 


.020 


.14 


.07 


.016 


.10 


.05 


.009 


.10 


.05 


.012 


.09 


.05 


.014 


.10 


.06 


.017 


.13 


.07 


.015 


.14 


.07 


.015 


• 20 


: 20 


20 


13 


20 : 


22 




: 20 


24 


14 


: 20 


: 23 


12 


20 : 


23 









.10 


.05 


.007 


.10 


.05 


.010 


.10 


.05 


.009 


.09 


.04 


.009 


.09 


.05 


.008 


.10 


.04 


.007 


.10 


.05 


.010 


.10 


.05 


.009 


.09 


.04 


.010 


.09 


.04 


.008 


.09 


.05 


.007 


.10 


.05 


.010 


.11 


.05 


.008 


.10 


.05 


.011 


.10 


.05 


.009 


.09 


.04 


.007 


.10 


.05 


.010 


.10 


.05 


.010 


.09 


.04 


.010 


.09 


.05 


.009 


.10 


.05 


.007 


.09 


.05 


.010 


.10 


.05 


.010 


.09 


.04 


.010 


.09 


.05 


.010 


.10 


.05 


.008 


.10 


.05 


.010 


.10 


.05 


.008 


-.10 


.05 


.011 


.09 


.05 


.009 


.10 


.04 


.007 


.10 


.05 


.010 


.10 


.05 


.008 


.09 


.05 


.0<0 


.09 


.05 


.009 



.10 


.05 


.008 


.09 


.04 


.011 


.10 


.05 


.009 


.10 


.05 


.009 


.10 


.05 


.013 


.10 


.05 


.007 


.09 


.04 


.011 


.10 


.05 


.010 


.09 


.05 


.010 


.10 


.05 


.012 


.10 


.05 


.007 


.09 


.05 


.011 


.10 


.05 


.011 


.10 


.05 


.009 


.10 


.05 


.013 


.10 


.05 


.010 


.09 


.04 


.011 


.10 


.05 


.010 


.09 


.05 


.010 


.09 


.05 


.012 


.10 


.05 


.009 


.09 


.04 


.009 


.09 


.05 


.010 


.09 


.04 


.009 


.09 


.05 


.012 


.10 


.06 


.008 


.09 


.04 


.011 


.10 


.05 


.011 


.10 


.05 


.009 


.10 


.05 


.013 


.10 


.05 


.005 


.09 


.04 


.011 


.10 


.05 


.010 


.10 


.05 


.010 


.10 


.05 


.013 



.09 


.05 


.009 


.10 


.05 


.011 


.11 


.06 


.011 


.12 


.06 


.016 


.12 


.06 


.014 


.09 


.05 


.010 


.10 


.05 


.011 


.11 


.05 


.010 


.11 


.06 


.018 


.12 


.06 


.013 


.09 


.05 


.009 


.11 


.05 


.011 


.11 


.05 


.010 


.1? 


.06 


.016 


.12 


.06 


.015 


.09 


.05 


.010 


.11 


.05 


.012 


.11 


.05 


.012 


.11 


.06 


.015 


.12 


.08 


.014 


.09 


.05 


.009 


.10 


.05 


.011 


.10 


.05 


.013 


.11 


.06 


.013 


.12 


.06 


.014 


.09 


.05 


.009 


.11 


.05 


.010 


.1 1 


.05 


.010 


.12 


.06 


.015 


.12 


.06 


.015 


.09 


.05 


.010 


.11 


.05 


.011 


.11 


.05 


.010 


.12 


.06 


.013 


.1? 


.06 


.014 



Empncei dittibuton found tigniicaniy different from the theoretical P datrbubon at tie p < .05 level 
Empncei detibutcn found wqnitcenty {Afferent from the theoretical F detributfon at tie p < .01 level 



ER J C - ^ 2 BEST COPY AVAILABLE 



Effects of Violations of Assumptions When Using ANOVA and ANCOVA With Unequal n's 1 1 



difference between the largest and smallest groups. Furthermore, even the ANCOVA analyses 
involving unequal slopes deviated from nominal alpha by only ±.02. At nominal alpha levels of 
a < .05 and a < .01 , most observed alpha levels were within ±.01 . 

Effects of Data Set Assumptions Wh e n Heterogeneity of Variance Is Present and Total n s 9Q 

Effects of Heterogeneity of Variances} in thft Bafoncod design : 30 : 30 ; 30 

When three groups of 30 are used, KS tests reveal no statistically significant differences 
between the tall regions of the empirical sampling distributions created in the simulation process 
and the theoretical F distribution - even in ANCOVA simulations when heterogeneity of variance 
(ratios of 1 :3:5) is joined with unequal regression slopes. The observed significance levels are all 
either equal to or within +..01 of the appropriate nominal alpha level. This, of course, suggests that 
in the balanced design the true risk of making type I error is equal to or very near the risk that the 
researcher believes it to be (see tables 2, 3 and 4). 

Effects of Heterogeneity of Variance^ in the Design : 28 : 30 : 32 

In these analyses the difference between the smallest and largest group sizes is 12.5%. 
With such close group sizes, KS testing reveals no significant differences from theoretical F. In 
most of the simulations, actual significance levels are within ±.01 of the intended significance 
level. However, observed significance levels are off by +.02 in some simulations - primarily 
condition B ANCOVA simulations having extreme heterogeneity of variances (one difference is 
+.03). Condition B ANCOVA analyses, as one will recall, are those where the smallest group size 
and slope are paired with the largest group variance, while oondition A ANCOVA analyses 
matched the smallest group size and slope with the smallest variance(see tables 2, 3 and 4). 

Effects of Heterogeneity o f Variances in the Design : 26 : 30 : 34 

In these analyses the difference between the largest and smallest group is 24%. With 
group differences of this magnitude, statistically significant differences between the empirical and 
theoretical sampling distributions begin to emerge- although they are confined to Condition B 
ANCOVAs when both extreme heterogeneity of variances and unequal slopes are present. 

For condition A at the nominal a < .10 level of significance, the observed significance 
levels range between .08 and .10. At nominal level of a < .05, most observed levels are between 
.04 and .06. At the nominal level of a < .01, the observed levels fall between .008 and .013. 

More variation Is found among condition B simulations, where at the nominal level of 
a < .1 0, observed significance levels fall between .10 and .12. At the nominal level of a < .05, 
observed significance level fall between .06 and .07, while at the nominal level of a < .01 the 
actual significance levels fall between .017 and .029 (see tables 2, 3 and 4). 
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Tabla 2: Actual Signlficgnoa Lavala Wh«ri Using the Nominal Stgnlfioanca of p < .10 and Total n = 90 



Group Size Condtion A 
Group Size Condtion B 

Largest to Smallest 



30 : 30 : 30 
30 : 30 : 30 



1:2 1 :3 .Ll5 



23 : 30 : 32 
32 : 30 : 2t 



26 : 30 : 34 
34 : 30 : 26 



1:2 1:3 1:5 1:2 1:3 1 :5 



24 : 30 : 36 
36 : 30 : 24 



1:2 1:3 1:5 



22 : 30 : 36 
36 : 30 : 22 



20 : 30 : 40 
40 : 30 : 20 



ANOVA 



CONDITION A 



Distribution*! Shape 

Normal 
Piatykurbc 
Leptokurbc 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 



ANOVA 



CONDITION B 



.09 


.10 


.10 


.10 


.10 


.10 


.09 


.09 


.09 


.08 


.08 


.07 


.08 


.08 


.07 


.09 


.08 


.07 


.09 


.09 


.10 


.10 


.10 


.10 


.08 


.08 


.09 


.08 


.08 


.08 


.08 


.07 


.07 


.08 


.07 


.07 


.09 


.10 


.10 


.10 


.10 


.11 


.06 


.08 


.09 


.08 


.07 


.07 


.09 


.08 


.07 


.08 


.06 


.07 


.09 


.10 


.10 


.10 


.10 


.10 


.09 


.09 


.09 


.08 


.08 


.07 


.08 


.07 


.07 


.08 


.07 


.07 


.09 


.09 


.10 


.10 


.10 


.10 


.09 


.08 


.08 


.08 


.08 


.08 


.08 


.08 


.07 


.08 


.07 


.07 


.10 


.10 


.10 


.10 


.10 


.10 


.10 


.09 


.09 


.08 


.07 


.07 


.09 


.08 


.07 


.09 


.08 


.07 


.10 
B 


.10 


.10 


.10 


.10 


.10 


.09 


.09 


.09 


.08 


.08 


.07 


.09 


.08 


.07 


.09 


.08 


.07 



.10 


.10 


.10 


.11 


.11 


.11 


.10 


.11 


.11 


.12 


.13 


.14 


.12 


.13 


.15 


.12 


.15 


.17 


.10 


.10 


.10 


.11 


.11 


.12 


.10 


.11 


.11 


.12 


.13 


.14 


.12 


.14 


.14 


.12 


.15 


.17 


.10 


.10 


.11 


.11 


.11 


.12 


.10 


.11 


.12 


.12 


.13 


.15 


.12 


.14 


.16 


.13 


.15 


.17 


.10 


.10 


.10 


.11 


.11 


.12 


.10 


.11 


.11 


.12 


.13 


.14 


.12 


.13 


.14 


.13 


.15 


.17 


.09 


.09 


.10 


.10 


.11 


.11 


.10 


.10 


.11 


.12 


.13 


.14 


.12 


.13 


.14 


.13 


.15 


.17 


.10 


.11 


.10 


.11 


.11 


.12 


.11 


.11 


.12 


.13 


.14 


.15 


.13 


.14 


.15 


.13 


.15 


.17 


.10 


.10 


.11 


.11 


.11 


.12 


.11 


.12 


.12 


.12 


.13 


.14 


.12 


.14 


.15 


.14 


.15 


.17 



DittribuUQntl Shtpt 

Norma! 
Piatykurbc 
Leptokurbc 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 



ANCOVA : CONDITION A, WITH EQUAL REGRESSION SLOPES 
Distribution*! Ship* 



Normal 
Piatykurbc 
Leptokurbc 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 



.10 


.10 


.10 


.09 


.09 


.09 


.09 


.09 


.08 


.08 


.08 


.08 


.07 


.07 


.06 


.07 


.07 


.06 


.10 


.10 


.10 


.09 


.09 


.09 


.09 


.09 


.08 


.08 


.08 


.08 


.07 


.07 


.06 


.07 


.06 


.06 


.10 


.11 


.11 


.09 


.09 


.09 


.09 


.09 


.08 


.08 


.08 


.08 


.07 


.06 


.06 


.07 


.07 


.06 


.09 


.10 


.11 


.09 


.09 


.09 


.09 


.09 


.08 


.08 


.08 


.09 


.07 


.06 


.06 


.07 


.07 


.06 


.09 


.10 


.10 


.09 


.09 


.09 


.06 


.09 


.08 


.09 


.08 


.08 


.07 


.00 


.06 


.07 


.06 


.06 


.10 


.10 


.11 


.10 


.09 


.09 


.09 


.08 


.08 


.08 


.08 


.08 


.07 


.07 


.07 


.07 


.07 


.07 


.10 


.10 


.11 


.09 


.09 


.10 


.09 


.08 


.08 


.08 


.08 


.08 


.07 


.07 


.06 


.07 


.07 


.06 



ANCOVA 



CONDITION B, WITH EQUAL REGRESSION SLOPES 



.10 


.10 


.10 


.10 


.11 


.12 


.11 


.11 


.12 


.13 


.13 


.13 


.13 


.14 


.15 


.14 


.16 


.17 


.10 


.10 


.10 


.10 


.11 


.11 


.11 


.11 


.12 


.12 


.13 


.14 


.12 


.14 


.15 


.14 


.16 


.17 


.10 


.10 


.10 


.10 


.11 


.12 


.12 


.12 


.12 


.12 


.13 


.14 


.13 


.14 


.15 


.14 


.16 


.17 


.11 


.11 


.10 


.11 


.11 


.13 


.12 


.12 


.13 


.13 


.14 


.14 


.13 


.14 


.15 


.14 


.16 


.18 


.10 


.10 


.10 


.10 


.11 


.12 


.12 


.13 


.13 


.12 


.13 


.14 


.12 


.14 


.15 


.14 


.16 


.18 


.10 


.10 


.11 


.11 


.11 


.12 


.10 


.11 


.12 


.12 


.13 


.14 


.12 


.14 


.15 


.14 


.16 


.17 


.10 


.10 


.11 


.11 


.11 


.12 


.10 


.11 


.12 


.12 


.13 


.14 


.13 


.14 


.15 


.14 


.17 


.18 



Distribution*} Snap* 

Normal 
Platykurtc 
Leptokurbc 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext. Skew & Lepto 

ANCOVA : CONDITION A, WITH UNEQUAL REGRESSION SLOPES 
Distributio n*} Shict 

Normal 
Piatykurbc 
Leptokurbc 
Moderate Skew 
Mod Skew & Platy 
Mod. SKew & Lepto 
Ext. Skew & Lepto 

ANCOVA : CONDITION B, WITH UNEQUAL REGRESSION SLOPES 

Distribution *} Shi pa 

Norma) 
Piatykurbc 
Leptokurbc 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew 4 Lepto 
Ext. Skew & Lepto 



.09 


.09 


.10 


.11 


.10 


.11 


.09 


.09 


.08 


.09 


.08 


.08 


.10 


.09 


.08 


.10 


.09 


.08 


.09 


.09 


.09 


.11 


.10 


.11 


.09 


.08 


.08 


.09 


.08 


.08 


.09 


.08 


.08 


.10 


.09 


.08 


.09 


.10 


.10 


.11 


.11 


.10 


.09 


.09 


.09 


.09 


.08 


.08 


.10 


.09 


.09 


.10 


.09 


.08 


.10 


.09 


.10 


.10 


.10 


.10 


.09 


.09 


.08 


.08 


.08 


.08 


.09 


.08 


.07 


.10 


.09 


.08 


.10 


.10 


.09 


.10 


.10 


.10 


.09 


.08 


.08 


.08 


.08 


.07 


.09 


.08 


.07 


.10 


.09 


.08 


.09 


.10 


.10 


.11 


.11 


.10 


.09 


.08 


.08 


.08 


.08 


.09 


.10 


.09 


.08 


.09 


.09 


.08 


.09 


.09 


.10 


.11 


.11 


.11 


.09 


.08 


.08 


.09 


.08 


.08 


.10 


.09 


.08 


.10 


.09 


.08 



.10 


.10 


.10 


.11 


.12 


.12 


.12 


.13 


.13 


.13 


.14 


.15 


.14 


.15 


.16 


.17 


.18 


.19 


.09 


.10 


.10 


.11 


.12 


.12 


.12 


.13 


.13 


.13 


.14 


.15 


.14 


.15 


.16 


.17 


.18 


.20 


.10 


.10 


.10 


.11 


.12 


.12 


.12 


.12 


.13 


.13 


.14 


.15 


.14 


.15 


.16 


.17 


.18 


.20 


.10 


.10 


.10 


.10 


.11 


.12 


.12 


.13 


.13 


.13 


.14 


.15 


.15 


.16 


.17 


.17 


.18 


.20 


.10 


.10 


.10 


.10 


.11 


.12 


.12 


.13 


.13 


.13 


.14 


.15 


.14 


.15 


.16 


.16 


.18 


.20 


.11 


.11 


.11 


.11 


.12 


.12 


.11 


.11 


.11 


.13 


.14 


.15 


.15 


.16 


.17 


.17 


.19 


.20 


.11 


.11 


.11 


.11 


.11 


.12 


.10 


.11 


.12 


.13 


.14 


.15 


.15 


.16 


.18 


.17 


.19 


.20 



Empncal dietibulon found eignitcanly drtterent from the theoretical F datnbuton at tie p < .06 level 
Empnc* detributon found eignitcanty afferent from tha theaetic* f dafrfcuton at tie p < .01 lava) 
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Tabl* 3: Actual Significant Ugtil Wh«n U«in 9 th a Nominal Si on if I can ct of D < .OS and Tota l n 5 M 



Group Sizes Condition A 
Group Sizes Condition B 

Largest to Smallest 
.Q fa u p V ariance Ratios 



30 : 30 : 30 
30 : 30 : 30 



1:2 1:3 1:6 



20 : 30 : 3S 
32 : 30 : 28 



20 : 30 : 34 
34 : 30 : 26 



24 : 30 : 3$ 
33 : 30 : 24 



22 : 30 : 30 
S3 : 30 : 22 



i:2 1:3 1:5 1:2 1:3 1:5 1:2 1:3 1:5 li2 Ll3 Ll5 



20 : 30 : 40 
40 : 30 : 20 



1L2 1:3 



ANOVA : CONDITION A 
DiMtributionMl $hapm 



Mormal 
Platykurbc 
Leptokurtic 
Moderate Skew 
Mod Skew&Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 

ANOVA : CONDITIO 
Distribution*} Ship* 

Normal 
Platykurtic 
Leptokurtic 
Moderate Skew 
Mod. Skew&Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 



.05 


.05 


.05 


.06 


.06 


.06 


.05 


.05 


.05 


.04 


.04 


.04 


.03 


.03 


.03 


.04 


.03 


.03 


.05 


.05 


.05 


.06 


.06 


.06 


.05 


.05 


.05 


.05 


.04 


.04 


.04 


.03 


.03 


.03 


.03 


.03 


.05 


.05 


.06 


.06 


.06 


.06 


.05 


.05 


.05 


.04 


.04 


.04 


.03 


.03 


.03 


.04 


.03 


.03 


.05 


.05 


.05 


.05 


.05 


.06 


.04 


.05 


.05 


.05 


.04 


.04 


.03 


.03 


.03 


.03 


.03 


.03 


.04 


.05 


.05 


.05 


.05 


.06 


.04 


.05 


.05 


.04 


.04 


.04 


.04 


.03 


.03 


.03 


.03 


.03 


.05 


.06 


.05 


.06 


.06 


.06 


.05 


.05 


.05 


.04 


.04 


.04 


.03 


.03 


.03 


.04 


.03 


.03 


.04 
B 


.05 


.05 


.06 


.06 


.06 


.05 


.05 


.05 


.04 


.04 


.04 


.04 


.03 


.03 


.03 


.03 


.03 



.05 


.06 


.06 


.06 


.06 


.07 


.06 


.06 


.07 


.07 


.07 


.09 


.06 


.07 


.09 


.07 


.08 


.10 


.05 


.05 


.06 


.06 


.06 


.06 


.06 


.06 


.07 


.06 


.07 


.08 


.06 


.07 


.OS 


.07 


.08 


.10 


.06 


.05 


.06 


.06 


.07 


.07 


.06 


.06 


.07 


.07 


.07 


.08 


.06 


.07 


.09 


.07 


.09 


.10 


.05 


.06 


.06 


.06 


.06 


.06 


.06 


.06 


.07 


.06 


.07 


.06 


.06 


.08 


.09 


.07 


.09 


.10 


.05 


.05 


.06 


.05 


.06 


.06 


.06 


.06 


.07 


.06 


.07 


.08 


.06 


.08 


.09 


.07 


.09 


.10 


.05 


.06 


.06 


.06 


.06 


.07 


.06 


.06 


.07 


.07 


.08 


.08 


.06 


.08 


.09 


.07 


.09 


.11 


.05 


.06 


.06 


.06 


.06 


.07 


.06 


.06 


.07 


.or 


.07 


.06 


.06 


.00 


.09 


.0/ 


.09 


.11 



ANCOVA 



CONDITION A, WITH EQUAL REGRESSION SLOPES 



.05 


.06 


.06 


.05 


.05 


.05 


.04 


.04 


.OA 


.04 


.04 


.04 


.03 


.03 


.03 


.03 


.03 


.03 


.05 


.06 


.06 


.05 


.05 


.05 


.04 


.04 


.04 


.04 


.04 


.04 


.03 


.03 


.03 


.03 


.03 


.03 


.05 


.05 


.06 


.05 


.05 


.05 


.04 


.04 


.04 


.04 


.04 


.04 


.03 


.03 


.03 


.03 


.03 


.03 


.06 


.06 


.06 


.05 


.05 


.05 


.04 


.04 


.05 


.04 


.04 


.04 


.03 


.03 


.03 


.03 


.03 


.03 


.05 


.06 


.05 


.05 


.05 


.05 


.04 


.04 


.05 


.04 


.04 


.04 


.03 


.03 


.02 


.03 


.03 


.03 


.05 


.05 


.06 


.05 


.04 


.05 


.04 


.04 


.04 


.04 


.04 


.04 


.03 


.03 


.03 


.03 


.03 


.03 


.06 


.06 


.06 


.05 


.05 


.05 


.04 


.04 


.04 


.04 


.04 


.04 


.03 


.03 


.03 


.03 


.03 


.03 



.05 


.05 


.05 


.06 


.06 


.07 


.06 


.07 


.08 


.07 


.07 


.09 


.06 


.08 


.09 


.07 


.09 


.11 


.05 


.05 


.05 


.06 


.06 


.06 


.06 


.07 


.08 


.07 


.08 


.08 


.06 


.07 


.09 


.07 


.09 


.11 


.06 


.06 


.06 


.06 


.06 


.07 


.06 


.06 


.07 


.06 


.07 


.08 


.07 


.08 


.09 


.07 


.09 


.11 


.05 


.05 


.05 


.06 


.06 


.06 


.06 


.07 


.07 


.07 


.07 


.08 


.06 


.07 


.09 


.07 


.09 


.10 


.05 


.05 


.06 


.06 


.06 


.06 


.06 


.06 


.07 


.07 


.08 


.06 


.06 


.07 


.09 


.06 


.08 


.10 


.06 


.06 


.06 


.06 


.06 


.06 


.05 


.06 


.07 


.07 


.07 


.08 


.06 


.07 


.09 


.08 


.09 


.11 


.06 


.06 


.06 


.06 


.06 


.07 


.06 


.06 


.07 


.07 


.08 


.08 


.06 


.07 


.09 


.06 


.09 


.11 



Dlmtributionml Ship* 

Normal 
Platykurbc 
Leptokurtic 
Moderate Skew 
Mod. Skew&Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 

ANCOVA : CONDITION 0, WITH EQUAL REGRESSION SLOPES 
Distribution*} Shapm 

Norma) 
Platykurbc 
Leptokurtic 
Moderate Skew 
Mod Skew&Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 

ANCOVA : CONDITION A, WITH UNEQUAL REGRESSION SLOPES 
Distribution^ Ship* 

Normal 
Platykurbc 
Leptokurtic 
Moderate Skew 
Mod Skew&Platy 
Mod. Skew & Lepto 
Ext. Skew & Lepto 

ANCOVA : CONDITION B, WITH UNEQUAL REGRESSION SLOPES 
DimtributJonal Ships 

Normal 
Platykurtic 
leptokurtic 
Moderate Skew 
Mod Skew&Platy 
Mod. Skew & Lepto 
Ext. Skew & Lepto 



.05 


.05 


.05 


.05 


.05 


.05 


.04 


.04 


.04 


.04 


.04 


.04 


.05 


.04 


.04 


.05 


.04 


.04 


.04 


.05 


.05 


.05 


.06 


.05 


.04 


.04 


.04 


.04 


.04 


.04 


.05 


.04 


.04 


.05 


.04 


.04 


.04 


.05 


.05 


.05 


.05 


.05 


.05 


.05 


.05 


.04 


.04 


.04 


.05 


.04 


.04 


.05 


.04 


.04 


.04 


.05 


.05 


.05 


.05 


.05 


.04 


.04 


.04 


.04 


.04 


.04 


.05 


.04 


.04 


.05 


.04 


.04 


.04 


.05. 


.05 


.05 


.05 


.05 


.04 


.04 


.04 


.04 


.04 


.04 


.05 


.04 


.04 


.05 


.04 


.04 


.04 


.05 


.05 


.05 


.05 


.05 


.04 


.04 


.04 


.04 


.04 


.04" 


.05 


.04 


.04 


.05 


.04 


.04 


.04 


.05 


.05 


.05 


.05 


.05 


.04 


.04 


.04 


.04 


.04 


.04 


.05 


.04 


.04 


.05 


.04 


.04 



.05 


.05 


.05 


.06 


.06 


.07 


.07 


.07 


.08 


.07 


.08 


.09 


.08 


.09 


.10 


.10 


.12 


.13 


.05 


.05 


.05 


.06 


.06 


.07 


.07 


.07 


.08 


.07 


.09 


.09 


.08 


.09 


.10 


.10 


.12 


.13 


.05 


.05 


.05 


.06 


.06 


.07 


.07 


.07 


.08 


.07 


.08 


.08 


.08 


.09 


.10 


.10 


.12 


.13 


.05 


.05 


.06 


.06 


.06 


.06 


.07 


.07 


.08 


.07 


.08 


.09 


.08 


.10 


.11 


.10. 


.12 


.13 


.05 


.06 


.06 


.07 


.07 


.07 


.07 


.07 


.08 


.07 


.08 


.08 


.08 


.09 


.10 


.10 


.11 


.13 


.05 


.05 


.06 


.06 


.06 


.07 


.06 


.06 


.07 


.07 


.08 


.06 


.08 


.09 


.10 


.11 


.12 


.13 


.05 


.06 


.06 


.06 


.06 


.07 


.06 


.06 


.07 


.07 


.08 


.09 


.06 


.09 


.11 


.10 


.11 


.13 



Empr ica) distibuton found wgnilicanly differ ent from the theoretical F dslnbubon at tie p < .05 level 
EmpricaJ dstibuton found •gnrtcanly different from the theoretical F dstrfcution at tie p < .01 level 
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T»bl« 4: Actual Signlfifianca Lavlg Whan Ua ing th# Nominal Siynificanct of D < .Q1 and Total n ■ M 



.009 


.009 


.012 


.013 


.014 


.013 


.009 


.010 


.011 


.008 


.007 


.008 


.006 


.005 


.006 


.006 


.007 


.007 


.008 


.009 


.013 


.012 


.012 


.013 


.009 


.010 


.010 


.009 


.007 


.008 


.005 


.005 


.005 


.009 


.007 


.007 


.009 


.009 


.01 1 


.014 


.015 


.016 


.009 


.011 


.01 1 


.008 


.008 


.008 


.008 


.007 


.007 


.007 


.006 


.006 


.008 


.010 


.01 1 


.014 


.015 


.017 


.009 


.009 


.011 


.009 


.008 


.009 


.006 


.006 


.006 


.008 


.008 


.008 


.009 


.009 


.009 


.013 


.016 


.016 


.008 


.009 


.011 


.007 


.007 


.009 


.006 


.005 


.005 


.009 


.008 


.007 


.009 


.009 


.010 


.016 


.015 


.016 


.011 


.012 


.012 


.007 .008 


.010 


.009 


.007 


.007 


.009 


.007 


.006 


.008 


.009 


.011 


.014 


.014 


.016 


.012 


.013 


.013 


.008 


.009 


.009 


.007 


.006 


.007 


.009 


.008 


.009 



.010 


.011 


.012 


.012 


.016 


.020 


.015 


.019 


.021 


.022 .025 


.031 


.014 


.019 


.027 


.022 


.029 


.037 


.010 


.011 


.012 


.012 


.014 


.020 


.016 


.017 


.021 


.022 . 028 


.030 


.013 


.018 


.028 


.022 


.029 


.034 


.010 


.010 


.012 


.013 


.015 


.016 


.014 


.017 


.020 


.021 .023 


.028 


.015 


.017 


.028 


.023 


.029 


.037 


.009 


.010 


.013 


.014 


.017 


.020 


.015 


.019 


.025 


.023 .028 


.032 


.014 


.020 


.027 


.023 


.030 


.039 


.009 


.011 


.013 


.014 


.018 


.020 


.017 


.019 


.023 


.023 .029 


.032 


.014 


.021 


.028 


.023 


.029 


.036 


.009 


.010 


.012 


.013 


.015 


.019 


.012 


.013 


.019 


.021 .025 


.029 


.014 


.019 


.023 


.024 


.027 


.035 


.010 


.010 


.013 


.013 


.014 


.019 


.011 


.013 


.017 


.024 .029 


.ted 


.015 


.019 


.024 


.023 


.027 


.034 



Group Sizes Condition A 30 ; 30 : 30 28 : 30 : 32 2« : 30 : 34 24 : 30 : 36 22 : 30 : 38 20 : 30 : 40 

Group Sizes Condition B 30 : 30 : 30 32 : 30 : 28 34 : 30 : ?§ 3* ; 30 : 24 38 : 30 : 22 40 : 30 : 20 

Largest to S mail est „ ^ „ ^ , m 

Grou p Variance Ratios 1:2 1:3 1:5 U2 U3 1:S 1:2 1:3 1:5 3j_2 1:3 1:5 Ll2 U3 1:5 1:2 1 : 3 US 

ANOVA : CONDITION A 
Dimtributionai Ship* 

Normal 
Platykurtic 
Leptokurbc 
Moderate Skew 
Mod. Skew & Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 

ANOVA : CONUTION B 
Dlmtributional Shapm 

Normal 
Platykurbc 
Leptokurbc 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 

ANCOVA : CONUTION A, WITH EQUAL REGRESSION SLOPES 

Oittributientl Ship* 

Normal 
Platykurtic 
Leptokurbc 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext. Skew & Lepto 

ANCOVA : CONDiTION B, WITH EQUAL REGRESSION SLOPES 

Orrtrfbtftfona/ Ship* 

Normal 
Platykurbc 
Leptokurbc 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 

ANCOVA : CONDITION A, WITH UNEQUAL REGRESSION SLOPES 
Distributio n*! Sh*a* 

Normal 
Platykurbc 
Leptokurbc 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 

ANCOVA : CONDITION B, WITH UNEQUAL REGRESSION SLOPES 



.017 


.018 


.018 


.008 


.008 


.011 


.011 


.011 


.01 1 


.010 


.009 


.009 


.009 


.008 


.008 


.010 


.008 


.008 


.017 


.016 


.018 


.007 


.009 


.011 


.010 


.010 


.012 


.009 


.009 


.010 


.009 


.008 


.008 


.010 


.008 


.007 


.017 


.017 


.017 


.007 


.009 


.010 


.010 


.008 


.009 


.008 


.008 


.008 


.008 


.007 


.008 


.009 


.009 


.006 


.016 


.016 


.016 


.008 


.008 


.011 


.010 


.011 


.010 


.011 


.009 


.008 


.009 


.007 


.009 


.010 


.009 


.008 


.014 


.015 


.015 


.004 


.006 


.010 


.010 


.011 


.011 


.010 


.009 


.009 


.007 


.008 


.009 


.009 


.007 


.006 


.019 


.020 


.017 


.008 


.009 


.010 


.011 


.011 


.008 


.009 


.009 


.009 


.008 


.007 


.009 


.010 


.008 


.007 


.017 


.019 


.018 


.008 


.009 


.010 


.012 


.011 


.010 


.010 


.011 


.010 


.006 


.007 


.008 


.008 


.008 


.007 



.010 


.011 


..011 


.013 


.017 


.020 


.013 


.016 


.020 


.021 .021 


.026 


.016 


.020 


.024 


.019 


.028 


.034 


.010 


.010 


.009 


.014 


.018 


.020 


.012 


.015 


.019 


.021 .022 


.027 


.016 


.019 


.022 


.021 


.027 


.033 


.010 


.012 


.012 


.013 


.013 


.017 


.013 


.017 


.018 


.020 .023 


.025 


.015 


.019 


.023 


.017 


.027 


.034 


.012 


.009 


.009 


.014 


.015 


.017 


.012 


.015 


.020 


.019 .020 


.027 


.014 


.020 


.022 


.015 


.028 


.037 


.010 


.011 


.011 


.014 


.017 


.018 


.012 


.016 


.021 


.016 .021 


.027 


.014 


.019 


.021 


.014 


.022 


.034 


.010 


.012 


.012 


.013 


.014 


.017 


.015 


.017 


.021 


.019 .020 


.025 


.015 


.021 


.024 


.017 


.026 


.034 


.011 


.010 


.012 


.013 


.013 


.016 


.013 


.015 


.020 


.017 .019 .025 


.015 


.020 


.02* 


.017 


.022 


.032 



.007 


.005 


.007 


.010 


.009 


.011 


.009 


.010 


.010 


.010 


.011 


.009 


.006 


.007 


.008 


.004 


.004 


.004 


.007 


.006 


.007 


.010 


.01 1 


.011 


.010 


.009 


.009 


.010 


.010 


.010 


.006 


.006 


.006 


.004 


.004 


.004 


.008 


.008 


.008 


.009 


.01 1 


.012 


.010 


.009 


.009 


.008 


.01 1 


.010 


.007 


.008 


.009 


.006 


.004 


.002 


.007 


.005 


.005 


.008 


.010 


.012 


.01 1 


.009 


.010 


.008 


.009 


.012 


.005 


.007 


.007 


.006 


.005 


.004 


.005 


.004 


.005 


.009 


.009 


.011 


.013 


.011 


.010 


.007 


.008 


.009 


.004 


.006 


.007 


.006 


.006 


.005 


.007 


.007 


.005 


.010 


.011 


.012 


.009 


.009 


.008 


.011 


.010 


.009 


.008 


.008 


.008 


.005 


.004 


.002 


.006 


.004 


.005 


.011 


.012 


.013 


.010 


.011 


.011 


.011 


.010 


.010 


.008 


.007 


.006 


.005 


.004 


.004 



.013 


.012 


.015 


.019 


.021 


.022 


.020 


.022 


.028 


.027 .030 


.033 


.023 


.028 


.033 


.028 


.039 


.046 


.011 


.012 


.016 


.018 


.020 


.023 


.019 


.023 


.026 


.027 .027 


.033 


.021 


.027 


.032 


.028 


.036 


.044 


.012 


.014 


.015 


.017 


.018 


.021 


.017 


.019 


.025 


.024 .030 


.032 


.020 


.029 


.033 


.029 


.037 


.048 


.012 


.013 


.016 


.017 


.018 


.022 


.018 


.022 


.029 


.024 .029 


.033 


.021 


.027 


.033 


.031 


.038 


.046 


.012 


.015 


.015 


.014 


.018 


.020 


.017 


.021 


.026 


.023 .026 


.031 


.022 


.028 


.034 


.025 


,032 


.044 


.013 


.012 


.015 


.018 


.019 


.021 


.014 


.019 


.024 


.025 .028 


.032 


.022 


.028 


.032 


.028 


.037 


.047 


.012 


.014 


.016 


.017 


.019 


.02! 


.013 


.016 


.021 


.023 .027 .030 


.022 


.028 


.035 


.029 


.037 


.047 



Norma) 
Platykurbc 
Leptokurbc 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew * Lepto 
bet Skew A Lepto 

Empmcal distibulon found aignilcanly drrfefsot from the theoretical F dstrfcubon at the p < .05 level 
Empiric* dstibuton found stgnitcanly different from the theoretical F datrtxrton at *t p < .01 level 
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pfterts of Hfttflrogflnfltty of Variant s In the Design : 24 : 30 : 36 

In these analyses the differences between the smallest and largest group is 33%. With 
differences of this magnitude, more statistically significant deviations from theoretical F emerge. 
Among ANOVA designs, condition B is most likely to emerge as significant. All but one condition 
B simulation having moderate heterogeneity of variance emerge as significant at the a < .05 level 
while simulations having extreme heterogeneity of variance emerge significant at the a < .01 
level. Among condition A ANOVA simulations, only three were significant at the a < .05 level. 

With ANCOVA, statistically significant differences from theoretical F are found only in 
condition B. There, nine equal slope simulations are significant at the a < .05 level and another 
six at the a < .01 level. When condition B is coupled with unequal slopes, all but one emerge 
statistically significant - most at the a < .01 level. 

At the nominal level of a < .10, the observed significance levels for condition A ANOVA 
and ANCOVA analyses fall between .07 and .09. At the nominal level of a < .05, observed levels 
of significance fall between .04 and .05, while at the nominal level of a < .01 the observed 
significance levels are between .007 and .012. With Condition B simulations at the nominal a < 
.10 level of significance, the observed significance levels range between .12 and .15. With the 
nominal level of a < .05, observed levels range between .06 and .09. Finally, with the nominal 
level of a < .01 , observed significance levels are between .016 and .033 (see tables 2, 3 and 4). 

Effects of Hfltflmgflneity of Variances in the Design : 22 : 3Q : 38 

In these analyses, the difference between the smallest and largest group is 42%. 
Statistically significant differences from theoretical F emerge with condition A ANCOVA having 
equal regression slopes (most at the a < .05 level), however only two condition A ANOVA 
simulations and none of the condition A ANCOVAs with unequal slopes are significantly different 
from the theoretical distribution. Differences are even more prevalent with condition B, where 
moderate and extreme heterogeneity of variance almost always emerge as significant at the 
a< .01 level. Among Condition B ANCOVAs with unequal slopes, even simulations with slight 
heterogeneity of group variances are significant at the a < .01 level. 

For condition A simulations at the nominal level of a < .10, observed significance levels 
are between .06 and .10, while they are between .02 and .05 for the nominal level of a < .05 and 
between .004 and .009 for a < .01 . For condition B the ranges are between .12 and .18 for 
a < .10, .06 and .1 1 for a < .05 and between .013 and .035 for a < .01 (see tables 2, 3 and 4). 

Effects ot Heterogeneity of Variances in the Das lon ; 20 ; 30 : 40 

In this set of analyses the difference between smallest and largest group Is 50%. 
Significant differences from theoretical F occur in almost all condition B simulations with even a 



ERIC 



17 
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slight degree of heterogeneity of variance (most at the a < .01 level). Condition A ANCOVA with 
equal slopes also produces many significant differences at least at the a < .05 level when coupled 
with heterogeneity of variances. Interestingly, almost no condition A ANOVA simulations are 
significant, nor were any of the condition A ANCOVA simulations having unequal slopes. 

For condition A simulations at the nominal level of a < .10, the observed significance 
levels fall between .06 and .10, while they are between .03 and .05 for nominal a < .05 and 
between .002 and .010 at the nominal level of a < .01 . For the nominal significance level of 
a < .10, the observed levels varied between .12 and .20, while for the nominal level of a < .01 
the observed levels varied between .014 and .048 (see tables 2, 3 and 4). 

Fffgrts nf nata Set Assumptions Whan Heteroge neity nf Variance is Present and Total n « 60 

Fffects of Heterogeneity nf Variances in the Balanced Design : 20 : 20 : 20 

When three groups of 20 were used, KS testing revealed no significant differences 
between tail regions of the theoretical and empirical sampling distributions. Also observed 
significance levels were close to their nominal counterparts (usually within ±.01 of nominal alpha). 
Again thto suggests that in the balanced designs the true risk of making type I error is equal to or 
near th< risk chosen by the researcher (see tables 5, 6 and 7). 

Fffects of Heterogeneity of Variances in the Design : 18 : 20 ; 22 

In these analyses, the difference between smallest and largest groups is 18%. This 
difference is so small that no significant differences from theoretical F appear. When nominal 
alpha is a < .1 0, the observed significance levels are within ±.01 for condition A ANOVA , 
condition A ANCOVA with unequal regression Slopes and condition B ANCOVA with equal 
regression slopes. Meanwhile, the observed significance levels are within ±.02 for condition B 
ANOVA, and condition A ANCOVA with equal slopes. Finally, observed levels vary between .1 1 
and .1 3 for condition B ANCOVA with unequal slopes. For the nominal alpha of a < .05, all 
observed levels varied within ±.01 except for condition B ANOVA with extreme variances and 
condition B ANCOVA with unequal slopes where differences may be +.02 (see tables 5, 6 and 7). 

Fffects of Heterogeneity n f Variances in the Design : 16 : 20 : 24 

In these analyses, the difference between the smallest and largest group size is 33%. 
Significant differences from theoretical F begin to appear when group sizes vary to this degree. 
Significant differences at the a < .05 level emerge in some condition A ANOVA's and condition A 
ANCOVA's with equal regression slopes and moderate or extreme heterogeneity of variances. 
Significant differences at either the a < .05 or a < .01 levels emerge in most condition B 
simulations having moderate or extreme heterogeneity of variances. 
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Table S: Actual Significance Levela Wh«n LUing th« Nominal Significance of p < .10 and Total n = 6Q 



Group Sizes Condition A 20 : 20 : 20 
Group Sizes Condition B 20 : 20 : 20 



Largest to Smallest 

2_Babofi 



18 : 20 : 22 
22 : 20 : 16 



.1:2 1:3 1 :5 



16 : 20 : 24 
24 : 20 : 1$ 



14 : 20 : 26 
26 : 20 : 14 



1:2 1:3 1:5 



12 : 20 : 26 
26 : 20 : 12 



UL2L- 



ANOVA : CONDITION A 
Dixtributionml Shmo* 

Norma! 
Piatykurbc 
Leptokurtic 
Moderate Skew 
Mod. Skew 4 Platy 
Mod. Skew & Lepto 
Ext. Skew & Lepto 



.09 


.09 


.09 


.09 


.09 


.09 


.08 


.07 


.07 


.07 


.06 


.06 


.07 


.06 


.05 


.09 


.09 


.09 


.09 


.09 


.09 


.08 


.07 


.07 


.07 


.06 


.06 


.07 


.06 


.05 


.09 


.09 


.10 


.10 


.09 


.09 


.08 


.08 


.08 


.07 


.07 


.06 


.07 


.06 


.06 


.09 


.09 


.10 


.09 


.09 


.10 


.08 


.07 


.07 


.07 


.06 


.06 


.07 


.06 


.05 


.09 


.09 


.10 


.09 


.09 


.09 


.07 


.07 


.07 


.06 


.06 


.06 


.07 


.06 


.05 


.10 


.09 


.10 


.10 


.09 


.09 


.08 


.03 


.07 


.08 


.07 


.06 


.07 


.06 


.05 


.10 


.10 


.10 


.10 


.10 


.10 


.08 


.08 


.08 


.08 


.07 


.06 


.07 


.06 


.05 



ANOVA : CONDITION B 
DiMtributionmi Shmo* 



Normal 
Platykurtic 
Leptokurbc 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 



.10 


.10 


.10 


.10 


.11 


.11 


.13 


.14 


.15 


.12 


.14 


.15 


.12 


.15 


.18 


.10 


.10 


.10 


.10 


.11 


.11 


.13 


.14 


.15 


.12 


.14 


.16 


.12 


.15 


.18 


.10 


.10 


.11 


.11 


.11 


.11 


.13 


.14 


.15 


.12 


.14 


.16 


.13 


.15 


.17 


.10 


.10 


.10 


.10 


.11 


.12 


.13 


.13 


.15 


.12 


.14 


.16 


.13 


.15 


.17 


.09 


.09 


.10 


.10 


.11 


.12 


.13 


.14 


.15 


.12 


.14 


.16 


.13 


.15 


.19 


.10 


.10 


.10 


.11 


.11 


.12 


.13 


.14 


.15 


.12 


.14 


.16 


.13 


.15 


.18 


.10 


.10 


.10 


.11 


.11 


.12 


.12 


.13 


.15 


.12 


.14 


.16 


.13 


.15 


.18 



ANCOVA : CONDITION A, WITH EQUAL REGRESSION SLOPES 

DiMtributionmi Ship* 

Norma! 
Platykurtic 
Leptokurtic 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 



.10 


.10 


.10 


.08 


.08 


.09 


.03 


.07 


.07 


.08 


.07 


.07 


.08 


.08 


.07 


.10 


.09 


.10 


.08 


.08 


.08 


.07 


.07 


.07 


.07 


.07 


.07 


.08 


.08 


.07 


.09 


.10 


.10 


.08 


.09 


.08 


,08 


.07 


.07 


.08 


.07 


,07 


.08 


.07 


.07 


.09 


.09 


.10 


.08 


.08 


.09 


.08 


.08 


.07 


.08 


.08 


.07 


.08 


.07 


,07 


.09 


.09 


.10 


.08 


.08 


.08 


.08 


.08 


.07 


.08 


.07 


.07 


.07 


.07 


.07 


.10 


.10 


.10 


.08 


.08 


.09 


.08 


.08 


.07 


.08 


.08 


.07 


.08 


.08 


.07 


.10 


.10 


.11 


.08 


.03 


.09 


.09 


.09 


.08 


.08 


.08 


.07 


.08 


.07 


.07 



ANCOVA : CONDITION B, WITH EQUAL REGRESSION SLOPES 

Attribution*] Ship* 

Normal 
Platykurtic 
Leptokurtic 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 



.10 


.10 


.10 


.10 


.11 


.11 


.12 


.13 


.13 


.12 


.13 


.15 


.14 


.15 


.18 


.10 


.10 


.10 


.10 


.10 


.11 


.12 


.13 


.13 


.12 


.13 


.15 


.13 


.15 


.18 


.10 


.10 


.10 


.10 


.11 


.11 


.12 


.13 


.14 


.13 


.13 


.15 


.14 


.15 


.17 


.10 


.10 


.10 


.10 


.11 


.11 


.12 


.13 


.13 


.13 


.14 


.15 


.13 


.15 


.18 


.10 


.10 


.10 


.11 


.11 


.11 


.12 


.13 


.14 


.13 


.15 


.15 


.13 


.16 


.18 


.10 


.10 


.10 


.11 


.11 


.11 


.13 


.14 


.14 


.13 


.14 


.15 


.13 


,15 


.17 


..10 


.10 


.10 


.10 


.11 


.11 


.13 


.13 


.14 


.13 


.14 


.16 


.13 


.15 


.17 



ANCOVA : CONDITION A, 
DiutributionMl Ship* 



Normal 
Platykurtic 
Leptokurtic 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext. Skew & Lepto 



WITH UNEQUAL REGRESSION SLOPES 



.09 


.09 


.09 


.10 


.09 


.09 


.09 


.08 


.08 


.09 


.09 


.08 


.09 


.08 


.07 


.09 


.09 


.09 


.10 


.09 


.09 


.09 


.08 


.07 


.09 


.09 


.08 


.09 


.08 


.07 


.09 


.09 


.10 


.10 


.10 


.09 


.09 


.08 


.07 


.10 


.09 


.09 


.09 


.08 


.07 


.09 


.09 


.09 


.10 


.09 


.09 


.08 


.08 


.08 


.09 


.09 


.08 


.09 


.08 


.07 


.09 


.09 


.09 


.09 


.09 


.09 


.08 


.07 


.07 


.09 


.08 


.07 


.08 


.07 


.07 


.09 


.09 


.10 


.10 


.09 


.10 


.C9 


.08 


.08 


.10 


.09 


.09 


.09 


.08 


.07 


.09 


.09 


.10 


.10 


.09 


.09 


.08 


.08 


.08 


.10 


.09 


.08 


.09 


.08 


.07 



ANCOVA 



Normal 
Platykurtic 
Leptokurtic 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 



CONDITION ft WITH UNEQUAL REGRESSION SLOPES 



.10 


.10 


.10 


.12 


.12 


.12 


,12 


.14 


.14 


.15 


.16 


.17 


.18 


.19 


.20 


.09 


.09 


.09 


.12 


.12 


.12 


.12 


.13 


.14 


.15 


.16 


.17 


.16 


.18 


.20 


.09 


.10 


.10 


.12 


.12 


.13 


.13 


.13 


.15 


.15 


.16 


.17 


.16 


.18 


.20 


.10 


.10 


.10 


.12 


.12 


.12 


.13 


.13 


.14 


.16 


.10 


.18 


,16 


.18 


.21 


.09 


.10 


.09 


.11 


.11 


.12 


.12 


.13 


.14 


.15 


.17 


.18 


.16 


.18 


.20 


.10 


.11 


.10 


.12 


.12 


.13 


.13 


.14 


.14 


.16 


.17 


.18 


.16 


.19 


.20 


.11 


.10 


.11 


.12 


.12 


.13 


,13 


.14 


.15 


.15 


.17 


.18 


.16 


.19 


.20 



Emprical datibuton found aignricanly different from the theoretical F dttrtoution at tie p < .05 level 
E anneal dttibuton found aignttcanty different from the theoretical F ditrtaufcon at tit p < .01 level 
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Group Sszes Condition A 
Group Sizes Condition B 

Smallest to Largest 
Q r ^ jp Variance Ffatl OS 



20 : 20 : 20 
20 : 20 : 20 



LL3 1:5 



18 : 20 : 22 
22 : 20 : 16 



J,;? LL3 US 



16 : 20 : 24 
24 : 20 : 16 



1;.? U3 V.5 



14 : 20 : 26 
26 : 20 : 14 



U_2 1 - 3 



12 : *0 : 28 
26 : 20 : 12 



1 ;? 1=3 1:5 



CONDITION A 



ANOVA : 

Normal 
Piatykurtic 
Leptokurtic 
Moderate Skew 
Mod. Skew* Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 



.05 
.05 
.04 
.04 
.04 
.04 



.05 .05 
.05 .05 



.05 
.04 

.04 

.05 



.04 .05 



ANOVA : CONDITION B 



Normal 
Piatykurtic 
Leptokurtic 
Moderate Skew 
Mod Skew* Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 



.05 
.05 
.05 
.05 
.05 
.05 
.05 



.05 
.05 
.05 
.05 
.05 
.05 
.06 



.05 
.05 
.05 
.05 
.05 



.06 
.06 
.05 
.06 
.06 
.06 
.06 



.04 
.04 
.04 
.04 
.04 
.04 
.04 



.04 .05 

.04 .05 

.04 .05 

.05 .05 

.05 .05 

.04 .05 

.05 .05 



.04 
.04 
.04 
.04 
.03 
.04 
.04 



.03 
.03 
.03 
.03 
.03 
.03 
.04 



.04 
.03 
.03 
.04 
.03 
.04 
.04 



.06 
.06 
.06 
.06 
.06 
.06 
.06 



.06 .06 

.06 .06 

.06 .07 

.06 .06 

.06 .06 

.06 .07 

.06 .06 



.07 
.07 
.07 
.06 
.07 
.07 
.07 



.08 
.08 
.08 
.08 
.08 
.08 
.08 



.OS 
.09 
.09 
.09 
.OS 
.09 
.OS 



ANCOVA : CONDITION A, 



WJTH EQUAL REGRESSION SLOPES 



Normal 
Piatykurtic 
Leptokurtic 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 



.05 
.05 
.05 
.05 
.05 
.05 
.05 



.05 
.05 
.05 
.06 
.05 
.05 
.06 



.06 
.06 
.06 
.06 
.05 
.06 
.06 



.04 
.04 
.04 
.04 
.04 
.04 
.04 



.04 .04 

.04 .04 

.04 .04 

.04 .04 

.04 .04 

.04 .04 

.04 .04 



.03 
.03 
.03 
.03 
.03 
.03 
.03 



.03 
.03 
.03 
.03 
.03 
.03 
.03 



.03 
.03 
.03 
.03 
.03 
.03 
.03 



.04 .03 .03 

.03 .03 .03 

.03 .03 .03 

.03 .03 .03 

.03 .03 .03 

.03 .03 .03 

.03 .03 .03 



.06 .07 .09 

.06 .07 .09 

.06 .07 .09 

.06 .08 .09 

.06 .07 .09 

.06 .08 .09 

.06 .08 .10 



.04 .04 .03 

.04 .04 .03 

.04 .03 .03 

.04 .03 .03 

.04 .03 .03 

.04 .03 .03 

.04 .03 .03 



.03 


.03 


.03 


.04 


.03 


.03 


.03 


.03 


.03 


.03 


.03 


.03 


.03 


.03 


.03 


.03 


.03 


.03 


.03 


.03 


.03 



.06 


.08 


.10 


.06 


.06 


.10 


.07 


.08 


.10 


.06 


.08 


.10 


.06 


.08 


.10 


.07 


.08 


.10 


.07 


.08 


.10 



.04 


.04 


.04 


.04 


.04 


.04 


.04 


.04 


.04 


.04 


.04 


.04 


.04 


.04 


.04 


.04 


.04 


.04 


.04 


.04 


.04 



ANCOVA 
ftfoFrfhuE/ona/ S/ttfit 

Norma} 
Piatykurtic 
Leptokurtic 
Moderate Skew 
Mod Skew & Platy 
Mod Skew & Lepto 
Ext Skew & Lepto 



CONDITION 0, WITH EQUAL REGRESSION SLOPES 



.05 


.05 


.05 


.05 


.05 


.05 


.05 


.06 


.06 


.05 


.05 


.05 


.05 


.05 


.05 


.05 


.05 


.05 


.06 


.06 


.06 



.06 
.06 
.06 
.05 
.05 
.05 
.05 



.06 
.06 
.06 
.06 
.06 
.06 



.06 
.06 
.06 
.06 
.06 
.06 



.05 .06 



.07 
.07 
.07 
.07 
.07 
.06 
.07 



.08 
.08 
.07 
.03 
.07 
.07 
.07 



.08 
.08 
.08 
.08 
.08 
.08 
.06 



ANCOVA 



CONDITION A, WITH UNEQUAL REGRESSION SLOPES 



.04 


.05 


.05 


.04 


.04 


.05 


.04 


.04 


.05 


.04 


.04 


.05 


.04 


.04 


.04 


.04 


.04 


.05 


.04 


.04 


.04 



Normal 
Piatykurtic 
Leptokurtic 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 

ANCOVA : COKUm* ft WITH UNEQUAL REGRESSION SLOPES 



.04 
.04 
.04 
.05 
.05 
.04 
.05 



.05 .05 
.05 .05 



.04 
.05 
.05 
.05 
.05 



.05 
.05 
.05 
.05 
.05 



.04 
.04 
.04 
.04 
.04 
.04 
.04 



.04 
.04 
.05 
.04 
.04 
.04 
.04 



.04 
.04 
.04 
.04 
.04 
.04 
.04 



.07 .08 -09 

.07 .09 .09 

.07 .08 .09 

.06 .08 .09 

.06 .08 .09 

.06 .03 .09 

.06 .08 .09 



.05 .04 .04 

.04 . 04 .04 

.05 . 04 .04 

.04 . 04 . 04 

.04 .04 .04 

.05 . 04 .04 

.05 .04 . 04 



.07 
.07 
.07 
.07 
.07 
.07 
.07 



.09 
.08 
.08 
.09 
-08 
.08 
.08 



.10 
.10 
.10 
.10 
.10 
.10 

.11 



.04 
.04 
.04 
.04 
.04 
.04 
.04 



.03 
.03 
.04 
.03 
.04 
.03 
.03 



.03 
.03 
.03 
.03 
.03 
.03 
.03 



Normal 
Piatykurtic 
Leptokurtic 
Moderate Skew 
Mod 8kew* Platy 
Mod Skew & Lepto 
Ext Skew & Lepto 



.05 


.05 


.05 


.05 


.05 


.05 


.05 


.05 


.05 


.06 


.05 


.06 


.05 


.05 




.05 


.05 


.05 


.05 


.06 


.06 



.06 


07 


.07 


.06 


.07 


.07 


.06 


.06 


.07 


.06 


.06 


.07 


.06 


.07 


.06 


.06 


.07 


.07 


.07 


.07 


.07 



.07 
.07 
.07 
.07 
.07 
.07 
.07 



.08 
.08 
.07 
.08 
.08 
.07 
.08 



.08 
.08 
.09 
.09 
.09 
.06 
.06 



.08 


.10 


.11 


.08 


.10 


.11 


.08 


.10 


.11 


.09 


.10 


.11 


.08 


.10 


.11 


.09 


.10 


.11 


.09 


.11 


.12 



.09 


.11 


.13 


.09 


.11 


.13 


.09 


.11 


.13 


.10 


.11 


.12 


.09 


.11 


.13 


.10 


.11 


.13 


.10 


.11 


.13 
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Table 7: A ctual Significant Lev el* Wh»n Ualno jfal Nom inal Sionificanoa of p < .01 and Total n « 60 



Group Sizeo Condition A 
Group Sizes Condition 8 

Largest to Smallest 
Group Variance Ratios 



20 : 20 : 20 
20 : 20 : 20 



1:2 1:3 



18 : 20 : 22 
22 : 20 : 18 



LL2 1:3 1:5 



16 : 20 : 24 
24 : 20 : 16 



1:2 1:3 1:5 



14 : 20 : 26 
26 : 20 : 14 



1:3 1:3 1:5 



12 : 20 : 28 
28 : 20 : \2 



JU.2 1;3 US 



4N0VA : CONDITION A 
DiMtributional Ship* 



Norma) 
Platy kurtic 
Leptokurlic 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 



.008 


.008 


.on 


.010 


.010 


.010 


.008 


.006 


.006 


.005 


.004 


.004 


.007 


.006 


.006 


.008 


.008 


.011 


.010 


.011 


.010 


.006 


.007 


.006 


.005 .004 


.004 


.006 


.005 


.006 


.007 


.008 


.009 


.009 


.009 


.011 


.007 


.005 


.005 


.005 


.005 


.003 


.008 


.006 


.006 


.008 


.009 


.010 


.008 


.010 


.011 


.005 


.006 


.007 


.005 


.004 


.004 


.007 


.006 


.005 


.007 


.008 


.008 


.007 


.010 


.013 


.007 


.006 


.007 


.005 


.004 


.003 


.006 


.005 


.005 


.007 


.008 


.010 


.008 


.008 


.010 


.005 


.005 


.005 


.005 


.004 


.004 


.007 


.005 


.005 


.007 


.008 


.010 


.007 


.007 


.013 


.006 


.005 


.005 


.005 


.004 


.005 


.006 


.007 


.005 



.009 


.009 


.011 


.014 


.014 


.016 


.018 


.022 


.031 


.020 


.024 


.031 


.019 


.022 


.031 


.009 


.009 


.011 


.013 


.014 


.016 


.019 


.023 


.033 


.0«.3 


.023 


.030 


.020 


.023 


.031 


.008 


.008 


.010 


.015 


.016 


.016 


.015 


.021 


.029 


.020 


.024 


.026 


.015 


.021 


.027 


.009 


.010 


.010 


.015 


.016 


.017 


.019 


.024 


.029 


.018 .023 


.029 


.019 


.023 


.031 


.009 


.011 


.011 


.015 


.016 


.017 


.021 


.024 


.028 


.020 


.023 


.026 


.018 


.022 


.033 


.008 


.009 


.011 


.014 


.014 


.017 


.017 


.022 


.028 


.020 


.023 


.029 


.017 


.022 


.029 


.008 


.009 


.010 


.014 


.016 


.018 


.017 


.023 


.028 


.019 


.022 


.027 


.019 


.025 


.031 



.017 


.016 


.016 


.008 


.009 


.011 


.007 


.007 


.006 


.009 .006 


.005 


.01 1 


.008 


.007 


.015 


.015 


.017 


.008 


.008 .010 


.008 


.007 


.007 


.008 .005 


.005 


.010 


.009 


.007 


.015 


.015 


.018 


.007 


.007 
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ANOVA : CONDITION B 



Normal 
Platykurtc 
Leptokurlic 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 



ANCOVA : CONDITION A, WITH EQUAL REGRESSION SLOPES 

Dlatrifruhanal Snap* 

Normal 
Platykurtc 
Leptokurlic 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext. Skew & Lepto 

ANCOVA : CONDITION B, WITH EQUAL REGRESSION SLOPES 
Distribution*! Shapa 

Normal 
Platykurlic 
Leptokurlic 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext Skew & Lepto 

ANCOVA : CONDITION A, WITH UNEQUAL REGRESSION SLOPES 
Distributional Shapa 

Normal 
Platykurtc 
Leptokurlic 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & Lepto 
Ext. Skew & Lepto 

ANCOVA : CONDITION B, WITH UNEQUAL REGRESSION SLOPES 

DiMtribution al Shape 

Normal 
Platykurtic 
Leptokurlic 
Moderate Skew 
Mod Skew & Platy 
Mod. Skew & lepto 
Ext Skew & Lepto 

^ EmpuicaJ ditfributon found aignifcanly different from the theaeticai F dstnbubon at the p < .05 level 
Empirical dttibuton found aip/tilcanlY different from the theoretical F datobufton at tit p < .01 lave) 
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At the nominal significance level of a < 10, observed levels of significance for condition A 
analyses are between .07 and .09, while for condition B they are between .12 and .15. For 
nominal level a < .05, they vary between .03 and .05 for condition A, and between .07 and .09 
for condition B. Finally, for the nominal level of a < .01 , the observed levels vary between .003 
and .008 for condition A and between .013 and .033 for condition B (see tables 5, 6 and 7). 

Effects of Heterogeneity of Variances in the Design : 14 : 20 : 26 

In these analyses, the difference between the smallest and largest group size is 46%. 
Group differences of this degree result in many significant differences between the empirical 
distributions and theoretical F. These differences (most at the a < .01 level) are found in both 
condition A and B ANOVA and the ANCOVA with unequal slopes. Interestingly, almost no 
significant differences emerge with condition A ANCOVA. 

At the nominal significance level of a < 10, observed levels of significance for condition A 
analyses are between .06 and .10, while for condition B they range between .12 and .18. For 
nominal level a < .05, they vary between .03 and .05 for condition A, and between .06 and .12 
for condition B. Finally, for the nominal level of a < .01 , observed levels vary between .003 and 
.010 for condition A and between .013 and .040 for condition B (see tables 5, 6 and 7). 

E ffects of Heterogeneity of Variance in the neainn : 12 : 20 : 28 

The difference between the largest and smallest groups in these analyses is 57%. 
Significant differences between the empirical and theoretical F distributions occur at the < .01 
level for every ANOVA and condition B ANCOVA that include moderate or extreme heterogeneity 
of variances. Only a few significant differences emerged among the condition A ANCOVA's ( a < 
.05 level) ; these were usually when extreme heterogeneity of variances was involved. 

At the nominal level of a < 10, observed levels of significance for condition A analyses are 
between .05 and .09, while for condition B they range between .12 and .21 . For nominal level 
a < .05, they vary between .03 and .04 for condition A, and between .06 and .13 for condition B. 
Finally, for the nominal level of a < .01 , observed levels of significance varied between .005 and 
.01 1 for condition A and between .015 and .047 for condition B (see tables 5, 6 and 7). 

A Fl P* 1 1 ™ k at the Simu lation Results: A Graphical Analysis 
Following significance testing aod examination of the actual alpha levels, 274 graphs were 
drawn which allowed a visual comparison of the differences in observed type I errors that are the 
result of the different group size combinations. Space does not allow the printing of the graphs in 
this paper. However, inspection of these graphs revealed some interesting patterns that are 
masked when analyzing this data purely from an algebraic perspective. 
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Examining first the ANOVA and ANCOVA graphs with no heterogeneity of slopes, lines 
representing less than 20% difference between the smallest and largest group size tend to cling 
together, relatively close to the line representing nominal alpha for condition A analyses. With 
condition B analyses, those analyses representing group sizes of less than 25% difference tend 
to cling together as well, although they are not as dose to the line representing nominal alpha as 
the condition A lines for group n's less than 20% different. With both condition A and condition B, 
when group differences were over 40%, their lines fell far from nominal alpha. Interestingly, the 
graphs for ANOVA and the graphs for ANCOVA with equal slopes (for almost any specific 
combination of distributional shape and heterogeneity of variances) appear amazingly similar. 

With the ANCOVA graphs when slopes are unequal, however, condition A and B graphs 
produce patterns that are both unique to their respective conditions and different from one 
another as well. Lines representing the different group size combinations fell close together and 
crossed frequently in condition A graphs, suggesting that much of the variation pictured by the 
graphs is due only to chance variation. These same lines in the condition B graphs fell far from 
nominal alpha and were spread far from one another as well. 

Finally, in comparing the six sets of graphs (ANOVA's for condition A and B, ANCOVA's 
with equal slopes for conditions A and B, and ANCOVA's with unequal slopes for conditions A 
and B), It should be noted that the average distance of the lines representing group size are 
closest to nominal alpha in the condition A ANCOVA's with unequal slopes, while the average 
distance of those lines is farthest for the condition B ANCOVA's with unequal slopes. 

FINDINGS AND CONCLUSIONS 

The Complex Nature of the Effects of V iolations of Data Set Assumptions 

Previous research (e.g., Glass, Peckham and Sanders, 1972; Johnson and Rakow, 1993) 
suggests that the combination of unequal group variances, unequal group sizes, and (in 
ANCOVA) unequal group regression slopes pose the greatest threat to robustness. The results 
of the present study produce findings conslstant with previous literature: in those analyses 
having homogeneity of group variances, the only simulations that emerged as statistically 
significant from theoretical F were those which had both a large degree of difference in group n's 
(50% and 57% difference between the largest and smallest groups) and unequal regression 
slopes. Additionally, among those simulations with near equal n's (the balanced designs, as well 
as those with group differences of 12.5% and 18%) no significant differences emerged, even 
when heterogeneity of group variances and regression slopes were present. 

Interestingly, unequal sample sizes did not - in and of themselves - threaten the 
robustness of the ANOVA or ANCOVA results, even when these differences were extreme. 
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However, unequal n's leave one's research vulnerable to the effects of violations of data set 
assumptions, a risk that grows greater as the degree of differences in group sizes increase. 

The joint relationship among unequal group sizes, heterogeneity of group variances and 
heterogeneity of regression slopes - crucial to understanding the issues involved here - is a 
complex one. To fully understand this relationship, one must first understand the pairs of 

relationships involved. 

Looking first at the paired relationship between unequal n's and unequal group variances, 
when the smallest group is coupled with the smallest variance and the largest group coupled with 
the largest variance (i.e., a match between magnitude of group sizes and variances), there will 
result less type I errors than one would expect based on normal theory. However, when the 
smallest n is coupled with the laigest variance and the largest n is coupled with the smallest 
variance (i.e., an inverse relationship between magnitude of group sizes and variances), the result 
is more type I errors than expected. In regards to the paired relationship between unequal n's 
and unequal group regression coefficients, when the smallest n is coupled with the smallest r (i.e., 
a match between the magnitude of sample sizes and r), more type I errors than expected will 
emerge. However, when the smallest n is coupled with the largest r (an inverse relationship 
between magnitude of group sizes and r), there will be less type I errors than expected (e.g., 
Johnson and Rakow, 1993; McClaren, 1972; Shields, 1978). 

Two distintive patterns emerge as these paired relationships are logically extended to a 
more complex form. The first pattern (condition A in these simulations) occurs when a match 
between the magnitude of group sizes and variances is coupled with a match in the magnitude 
between group sizes and r. The former will increase the risk of type I error, while the latter will act 
to decrease type I errors. The net effect is a wash-out. This phenomenon explains the otherwise 
baffling pattern that emerged in the condition A ANCOVA simulations involving unequal slopes: 
even when extremely unequal group sizes were coupled with extreme heterogeneity of 
variances, statistically significant differences were almost nonexistant. Similarly, the observed 
significance levels were very close to the alpha levels posited by normal theory, even in face of 

these data set violations. 

The second pattern (condition B) occurs when the inverse coupling of group sizes and 
magnitude of r Is joined with the matched group sizes and variances. Both of these paired 
combinations will individually Increase the risk of type I error. When these two pairs of conditions 
are combined into a single, compound condition, the risks are additive - and the result is type I 
error rates inflated far beyond what either of the two pairs of conditions alone could have 
produced. This, of course, explains why the observed significance levels of the condition B 
ANCOVA analyses with unequal slopes in this study are so much higher than nominal alpha. 
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Actual vs Expected Type I Errors 
Typically, when a researcher is engaged in hypothesis testing, the first choice he or she 
will make is which statistical procedure, from the many available, will be used to do the testing. 
This decision is quickly followed by a second: which level of significance to use. The level of 
significance, of course, defines the maximum risk of making type i error that a researcher is willing 
to accept. When chosing a significance level, the researcher assumes - often on the basis of 
blind faith alone - that it represents the true risk of making such error. 

These results suggest that this faith may not always be well placed. Unequal group sizes 
alone do not seem to significantly affect type I error rates, however unequal n's coupled with 
heterogeneity of variances and/or heterogeneity of regression slopes often will. This, in turn, can 
lead to misleading or inaccurate interpretations of one's data. 

Besides simply documenting the results of the simulations in this study, the tables 
enclosed in this paper serve a second purpose: they offer the research practitioner whose study 
involves ANOVA or ANCOVA with three groups some idea of the true risk of type I error faced in 
their own research. This can be particularly useful in situations where the data has already been 
collected and, upon checking, a researcher finds one or more data set violations. Consulting the 
table that corresponds with the researcher's chosen nominal level of significance, the sample size 
combination that best reflects the researcher's own group sizes can be found. From there, the 
user of the table can locate the particular combination of violations found in his or her data set. 
The proportions found in the body of the table represent the actual levels of significance (i.e. the 
observed proportion of type I error made in these simulations) for the nominal significance levels 
of a < .1 0, a < .05 and a < .01 . While these proportions will not provide the exact risk of type I 
error for another data set, they should provide estimates more accurate than normal theory 
provides • at least when three groups with similar group sizes to those in the tables are used. 

A word of caution is called for. These tables should be helpful for those researchers 
using three groups with group size distributions similar to those included on the tables. However, 
the ability of these proportions to generalize to research not having three groups, or to group size 
distributions far different from those reflected in the table has not been established. Indeed, 
research by Tomarken and Serlin (1986) suggests that these proportions may not, in fact, 
generalize beyond the three group research model. Their research suggests that the effects of 
heterogeneity of group variances increases as the number of groups increases. 
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.On How Unequal Can Sample Sires Be Anyway? 
Based on these simulation results, the following general recommendations can be made 
for the researcher who uses ANOVA or ANCOVA to test for differences between thrts groups. 
First, although in theory unequal n's, in and of themselves, do not jeopardize robustness, in 
practice this will never be the case. The reason is that it is virtually impossible for a researcher to 
stumble into a research situation where the data contains no violations of statistical assumptions 
(Glass et al., 1972). Unequal n's will generally exaggerate the effects of any data set violations that 
do exist. 

So how unequal can group n's be? When the difference between the largest and 
smallest groups is no more than 24%, the results produce type I error rates almost as close to 
nominal alpha as the balanced (equal n) design. Differences between 25% and 33% seem to be 
marginally adequate if the smallest group has the smallest variance and the largest group has the 
largest variance; otherwise group size differences of this magnitude will significantly affect type I 
error rates. Finally, differences above 33% produce actual type I error rates too different from 
nominal alpha to be considered statistically robust. Samples this unequal should be avoided if at 
all possible. 
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